{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JAPoU8Sm5E6e"
   },
   "source": [
    "# Comparing Pipeline Runs with Vertex AI Experiments\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d220917f1302"
   },
   "source": [
    "## Learning objectives\n",
    "\n",
    "1. Formalize a training component.\n",
    "2. Build a training pipeline.\n",
    "3. Run several Pipeline jobs and log their results.\n",
    "4. Compare different Pipeline jobs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO"
   },
   "source": [
    "## Introduction\n",
    "In this notebook, you learn how to use `Vertex AI Experiments` to log a pipeline job and compare different pipeline jobs.\n",
    "\n",
    "Depending on the model life cycle of your data science team, you would like to experiment and track training Pipeline runs and its associated parameters. Then, you would to compare runs of these Pipelines to each others in order to figure out which is the best configuration generates the model you will register in the Vertex AI Model Registry."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "263933842022"
   },
   "source": [
    "### Dataset\n",
    "\n",
    "The dataset used for this tutorial is the [Iris dataset](https://www.tensorflow.org/datasets/catalog/iris) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). This dataset does not require any feature engineering. The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of Iris flower species from a class of three species: setosa, virginica, or versicolor.\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in this student lab notebook -- try to complete this notebook first and then review the [solution notebook](../solutions/comparing_pipeline_runs.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i7EUnXsZhAGF"
   },
   "source": [
    "### Install additional packages\n",
    "\n",
    "Install additional package dependencies not installed in your notebook environment, such as Vertex AI SDK and KFP SDK. Use the latest major GA version of each package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "2b4ef9b72d43"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m  WARNING: The scripts pyrsa-decrypt, pyrsa-encrypt, pyrsa-keygen, pyrsa-priv2pub, pyrsa-sign and pyrsa-verify are installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The script normalizer is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The script tb-gcp-uploader is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The script tabulate is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The script strip-hints is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The script jsonschema is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The scripts dsl-compile, dsl-compile-v2 and kfp are installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "# The Vertex AI Workbench Notebook product has specific requirements\n",
    "IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\")\n",
    "IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
    "    \"/opt/deeplearning/metadata/env_version\"\n",
    ")\n",
    "\n",
    "# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
    "USER_FLAG = \"\"\n",
    "if IS_WORKBENCH_NOTEBOOK:\n",
    "    USER_FLAG = \"--user\"\n",
    "\n",
    "!pip3 install {USER_FLAG} --force-reinstall 'google-cloud-aiplatform>=1.15' -q --no-warn-conflicts\n",
    "!pip3 install {USER_FLAG} kfp -q --no-warn-conflicts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hhq5zEbGg0XX"
   },
   "source": [
    "### Restart the kernel\n",
    "\n",
    "After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "EzrelQZ22IZj"
   },
   "outputs": [],
   "source": [
    "# Automatically restart kernel after installs\n",
    "import os\n",
    "\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lWEdiXsJg0XY"
   },
   "source": [
    "## Before you begin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WReHDGG5g0XY"
   },
   "source": [
    "### Set your project ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "3c8049930470"
   },
   "outputs": [],
   "source": [
    "PROJECT_ID = \"[your-project-ID]\"  # Replace with your Project ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "f2e3c0f2cbfb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n"
     ]
    }
   ],
   "source": [
    "! gcloud config set project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2aa333eca058"
   },
   "source": [
    "#### Region\n",
    "\n",
    "You can also change the `REGION` variable, which is used for operations\n",
    "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
    "\n",
    "- Americas: `us-central1`\n",
    "- Europe: `europe-west4`\n",
    "- Asia Pacific: `asia-east1`\n",
    "\n",
    "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
    "\n",
    "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "d8b34ef9a3d0"
   },
   "outputs": [],
   "source": [
    "REGION = \"[your-region]\"  # Replace with your region\n",
    "\n",
    "if REGION == \"[your-region]\":\n",
    "    REGION = \"us-central1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "126548a06aa1"
   },
   "source": [
    "#### UUID\n",
    "\n",
    "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "e660b8504e63"
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import string\n",
    "\n",
    "\n",
    "# Generate a uuid of a specifed length(default=8)\n",
    "def generate_uuid(length: int = 8) -> str:\n",
    "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
    "\n",
    "\n",
    "UUID = generate_uuid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zgPO1eR3CYjk"
   },
   "source": [
    "### Create a Cloud Storage bucket\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
    "\n",
    "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "MzGDU7TWdts_"
   },
   "outputs": [],
   "source": [
    "BUCKET_NAME = \"[your-bucket-name]\"  # Replace with your unique bucket name\n",
    "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-EcIXiGsCePi"
   },
   "source": [
    "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "NIq7R4HZCfIc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating gs://qwiklabs-gcp-00-1fe7a1835a07/...\n"
     ]
    }
   ],
   "source": [
    "# TODO 1\n",
    "# Create a regional Cloud Storage bucket\n",
    "# TODO 1: Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ucvCsknMCims"
   },
   "source": [
    "Finally, validate access to your Cloud Storage bucket by examining its contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "vhOb7YnwClBb"
   },
   "outputs": [],
   "source": [
    "! gsutil ls -al $BUCKET_URI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b7e24e522bee"
   },
   "source": [
    "#### Service Account\n",
    "\n",
    "**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "77b01a1fdbb4"
   },
   "outputs": [],
   "source": [
    "SERVICE_ACCOUNT = \"[your-service-account]\"  # Replace '[your-service-account]' with your project's Compute Engine service account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
     "id": "121d7ca29426"
   },
   "outputs": [],
   "source": [
     "import sys\n",
     "\n",
     "IS_COLAB = \"google.colab\" in sys.modules\n",
     "\n",
     "if (\n",
     "    SERVICE_ACCOUNT == \"\"\n",
     "    or SERVICE_ACCOUNT is None\n",
     "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
     "):\n",
     "    # Get your service account from gcloud\n",
     "    if not IS_COLAB:\n",
     "        shell_output = !gcloud auth list 2>/dev/null\n",
     "        SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
     "\n",
     "    if IS_COLAB:\n",
     "        shell_output = ! gcloud projects describe  $PROJECT_ID\n",
     "        project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n",
     "        SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
     "\n",
     "    print(\"Service Account:\", SERVICE_ACCOUNT)"
    ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aa175e2960ac"
   },
   "source": [
    "#### Set service account access for Vertex AI Pipelines\n",
    "\n",
    "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step -- you only need to run these once per service account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "f88cb0488c08"
   },
   "outputs": [],
   "source": [
    "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
    "\n",
    "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fXUqOdIaLbjf"
   },
   "source": [
    "### Download training dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "9fYX14c0LfmU"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying gs://cloud-samples-data/ai-platform/iris/classification/evaluate.csv [Content-Type=text/csv]...\n",
      "Copying gs://cloud-samples-data/ai-platform/iris/classification/train.csv [Content-Type=text/csv]...\n",
      "Copying gs://cloud-samples-data/ai-platform/iris/iris_data.csv [Content-Type=application/octet-stream]...\n",
      "Copying gs://cloud-samples-data/ai-platform/iris/iris_predict.csv [Content-Type=text/csv]...\n",
      "- [4 files][  5.4 KiB/  5.4 KiB]                                                \n",
      "==> NOTE: You are performing a sequence of gsutil operations that may\n",
      "run significantly faster if you instead use gsutil -m cp ... Please\n",
      "see the -m section under \"gsutil help options\" for further information\n",
      "about when gsutil -m can be advantageous.\n",
      "\n",
      "Copying gs://cloud-samples-data/ai-platform/iris/iris_target.csv [Content-Type=application/octet-stream]...\n",
      "Copying gs://cloud-samples-data/ai-platform/iris/iris_test.csv [Content-Type=text/csv]...\n",
      "Copying gs://cloud-samples-data/ai-platform/iris/iris_training.csv [Content-Type=text/csv]...\n",
      "| [7 files][  8.4 KiB/  8.4 KiB]                                                \n",
      "Operation completed over 7 objects/8.4 KiB.                                      \n"
     ]
    }
   ],
   "source": [
    "DATASET_URI = \"gs://cloud-samples-data/ai-platform/iris\"\n",
    "\n",
    "!gsutil cp -r $DATASET_URI $BUCKET_URI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XoEqT2Y4DJmf"
   },
   "source": [
    "### Import libraries and define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "pRUOFELefqf1"
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "# General\n",
    "import os\n",
    "import time\n",
    "\n",
    "logger = logging.getLogger(\"logger\")\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "\n",
    "import kfp.v2.compiler as compiler\n",
    "# Pipeline Experiments\n",
    "import kfp.v2.dsl as dsl\n",
    "# Vertex AI\n",
    "from google.cloud import aiplatform as vertex_ai\n",
    "from google.cloud.aiplatform_v1.types.pipeline_state import PipelineState\n",
    "from kfp.v2.dsl import Metrics, Model, Output, component"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "OAY0QKZD8qNP"
   },
   "outputs": [],
   "source": [
    "# Experiments\n",
    "TASK = \"classification\"\n",
    "MODEL_TYPE = \"xgboost\"\n",
    "EXPERIMENT_NAME = f\"{PROJECT_ID}-{TASK}-{MODEL_TYPE}-{UUID}\"\n",
    "\n",
    "# Pipeline\n",
    "PIPELINE_TEMPLATE_FILE = \"pipeline.json\"\n",
    "PIPELINE_URI = f\"{BUCKET_URI}/pipelines\"\n",
    "TRAIN_URI = f\"{BUCKET_URI}/iris/iris_data.csv\"\n",
    "LABEL_URI = f\"{BUCKET_URI}/iris/iris_target.csv\"\n",
    "MODEL_URI = f\"{BUCKET_URI}/model\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "inR70nh38PeK"
   },
   "source": [
    "### Initialize Vertex AI SDK for Python\n",
    "\n",
    "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "Nz0nasrh8T3c"
   },
   "outputs": [],
   "source": [
    "vertex_ai.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "container:training,prediction,xgboost"
   },
   "source": [
    "### Set pre-built containers\n",
    "\n",
    "Set the pre-built Docker container image for training and prediction.\n",
    "\n",
    "\n",
    "For the latest list, see [Pre-built containers for training](https://cloud.google.com/ai-platform-unified/docs/training/pre-built-containers).\n",
    "\n",
    "\n",
    "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "XujRA5ueox9U"
   },
   "outputs": [],
   "source": [
    "TRAIN_IMAGE = vertex_ai.helpers.get_prebuilt_prediction_container_uri(\n",
    "    framework=\"xgboost\", framework_version=\"1.1\", accelerator=\"cpu\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t1NLYz1R-KWv"
   },
   "source": [
    "## Formalize your training as pipeline component\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jnfKxpj0-Z0H"
   },
   "source": [
    "Before you start running your pipeline experiments, you have to formalize your training as pipeline component.\n",
    "\n",
    "To do that, you build the pipeline by using the `kfp.v2.dsl.component` decorator to convert your training task into a pipeline component. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "jv_-vU46_eFN"
   },
   "outputs": [],
   "source": [
    "@component(\n",
    "    packages_to_install=[\n",
    "        \"numpy==1.18.5\",\n",
    "        \"pandas==1.0.4\",\n",
    "        \"scikit-learn==0.23.1\",\n",
    "        \"xgboost==1.1.1\",\n",
    "    ]\n",
    ")\n",
    "def custom_trainer(\n",
    "    train_uri: str,\n",
    "    label_uri: str,\n",
    "    max_depth: int,\n",
    "    learning_rate: float,\n",
    "    boost_rounds: int,\n",
    "    model_uri: str,\n",
    "    metrics: Output[Metrics],\n",
    "    model_metadata: Output[Model],\n",
    "):\n",
    "\n",
    "    # import libraries\n",
    "    import logging\n",
    "    import uuid\n",
    "    from pathlib import Path as path\n",
    "\n",
    "    import pandas as pd\n",
    "    import xgboost as xgb\n",
    "    from sklearn.metrics import accuracy_score\n",
    "    from sklearn.model_selection import train_test_split\n",
    "\n",
    "    # variables\n",
    "    gs_prefix = \"gs://\"\n",
    "    gcsfuse_prefix = \"/gcs/\"\n",
    "    train_path = train_uri.replace(gs_prefix, gcsfuse_prefix)\n",
    "    label_path = label_uri.replace(gs_prefix, gcsfuse_prefix)\n",
    "    model_path = model_uri.replace(gs_prefix, gcsfuse_prefix)\n",
    "\n",
    "    def get_logger():\n",
    "        \"\"\"\n",
    "        Get the logger\n",
    "        \"\"\"\n",
    "        logger = logging.getLogger(__name__)\n",
    "        logger.setLevel(logging.INFO)\n",
    "        handler = logging.StreamHandler()\n",
    "        handler.setFormatter(\n",
    "            logging.Formatter(\"%(asctime)s - %(name)s - %(levelname)s - %(message)s\")\n",
    "        )\n",
    "        logger.addHandler(handler)\n",
    "        return logger\n",
    "\n",
    "    def get_data(\n",
    "        train_path: str, label_path: str\n",
    "    ) -> (xgb.DMatrix, pd.DataFrame, pd.DataFrame):\n",
    "        \"\"\"\n",
    "        Get the data\n",
    "        Args:\n",
    "            train_path: the path of the train data\n",
    "            label_path: the path of the label data\n",
    "        Returns:\n",
    "            the train data and the label data\n",
    "        \"\"\"\n",
    "        # Load data into pandas, then use `.values` to get NumPy arrays\n",
    "        data = pd.read_csv(train_path).values\n",
    "        labels = pd.read_csv(label_path).values\n",
    "\n",
    "        # Convert one-column 2D array into 1D array for use with XGBoost\n",
    "        labels = labels.reshape((labels.size,))\n",
    "        train_data, test_data, train_labels, test_labels = train_test_split(\n",
    "            data, labels, test_size=0.2, random_state=7\n",
    "        )\n",
    "\n",
    "        # Load data into DMatrix object\n",
    "        dtrain = xgb.DMatrix(train_data, label=train_labels)\n",
    "        return dtrain, test_data, test_labels\n",
    "\n",
    "    def train_model(max_depth: int, eta: int, boost_rounds, dtrain: xgb.DMatrix):\n",
    "        \"\"\"\n",
    "        Train the model\n",
    "        Args:\n",
    "            max_depth: the max depth of the model\n",
    "            eta: the eta of the model\n",
    "            boost_rounds: the boost rounds of the model\n",
    "            dtrain: the train data\n",
    "        Returns:\n",
    "            the trained model\n",
    "        \"\"\"\n",
    "        # Train XGBoost model\n",
    "        param = {\"max_depth\": max_depth, \"eta\": eta}\n",
    "        model = xgb.train(param, dtrain, num_boost_round=boost_rounds)\n",
    "        return model\n",
    "\n",
    "    def evaluate_model(model, test_data, test_labels):\n",
    "        \"\"\"\n",
    "        Evaluate the model\n",
    "        Args:\n",
    "            model: the trained model\n",
    "            test_data: the test data\n",
    "            test_labels: the test labels\n",
    "        Returns:\n",
    "            the accuracy of the model\n",
    "        \"\"\"\n",
    "        dtest = xgb.DMatrix(test_data)\n",
    "        pred = model.predict(dtest)\n",
    "        predictions = [round(value) for value in pred]\n",
    "        # Evaluate predictions\n",
    "        accuracy = accuracy_score(test_labels, predictions)\n",
    "        return accuracy\n",
    "\n",
    "    def save_model(model, model_path):\n",
    "        \"\"\"\n",
    "        Save the model\n",
    "        Args:\n",
    "            model: the trained model\n",
    "            model_path: the path of the model\n",
    "        \"\"\"\n",
    "        model_id = str(uuid.uuid1())\n",
    "        model_path = f\"{model_path}/{model_id}/model.bst\"\n",
    "        path(model_path).parent.mkdir(parents=True, exist_ok=True)\n",
    "        model.save_model(model_path)\n",
    "\n",
    "    # Main ----------------------------------------------\n",
    "\n",
    "    dtrain, test_data, test_labels = get_data(train_path, label_path)\n",
    "    model = train_model(max_depth, learning_rate, boost_rounds, dtrain)\n",
    "    accuracy = evaluate_model(model, test_data, test_labels)\n",
    "    save_model(model, model_path)\n",
    "\n",
    "    # Metadata ------------------------------------------\n",
    "    metrics.log_metric(\"accurancy\", accuracy)\n",
    "    model_metadata.uri = model_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U1UiTZhkVoFM"
   },
   "source": [
    "## Build a pipeline\n",
    "\n",
    "Below code will perform creating pipelineJob in associated project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "9Gfr6pNLU-dB"
   },
   "outputs": [],
   "source": [
    "@dsl.pipeline(name=\"custom-training-pipeline\")\n",
    "def pipeline(\n",
    "    train_uri: str,\n",
    "    label_uri: str,\n",
    "    max_depth: int,\n",
    "    learning_rate: float,\n",
    "    boost_rounds: int,\n",
    "    model_uri: str,\n",
    "):\n",
    "\n",
    "    custom_trainer(\n",
    "        train_uri, label_uri, max_depth, learning_rate, boost_rounds, model_uri\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RkfZ7qVAVjBO"
   },
   "source": [
    "### Compile your pipeline into a JSON file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "oYlLBGUSVibG"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jupyter/.local/lib/python3.7/site-packages/kfp/v2/compiler/compiler.py:1293: FutureWarning: APIs imported from the v1 namespace (e.g. kfp.dsl, kfp.components, etc) will not be supported by the v2 compiler since v2.0.0\n",
      "  category=FutureWarning,\n"
     ]
    }
   ],
   "source": [
    "# TODO 2\n",
    "# Compile the pipeline with required parameters\n",
    "compiler.Compiler().compile(# TODO 2: Your code goes here)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "95vG4-zPWc0B"
   },
   "source": [
    "## Submit and track pipeline runs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZNb6kZ2l5t-O"
   },
   "source": [
    "### Submit Pipeline runs\n",
    "\n",
    "Now that you have the pipeline, you define its training configuration depending on the defined parameters. Below you have an example and how you can submit several pipeline runs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "XPy0Jc8xXgpa"
   },
   "outputs": [],
   "source": [
    "runs = [\n",
    "    {\"max_depth\": 4, \"learning_rate\": 0.2, \"boost_rounds\": 10},\n",
    "    {\"max_depth\": 5, \"learning_rate\": 0.3, \"boost_rounds\": 20},\n",
    "    {\"max_depth\": 3, \"learning_rate\": 0.1, \"boost_rounds\": 30},\n",
    "    {\"max_depth\": 6, \"learning_rate\": 0.5, \"boost_rounds\": 40},\n",
    "    {\"max_depth\": 5, \"learning_rate\": 0.4, \"boost_rounds\": 30},\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "G0hm1no_WY8o"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:Creating PipelineJob\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob created. Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105059\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob created. Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105059\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pipeline_job = aiplatform.PipelineJob.get('projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105059')\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:pipeline_job = aiplatform.PipelineJob.get('projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105059')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/custom-training-pipeline-20221014105059?project=21417957250\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/custom-training-pipeline-20221014105059?project=21417957250\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Associating projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105059 to Experiment: qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.metadata.experiment_resources:Associating projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105059 to Experiment: qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:Creating PipelineJob\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob created. Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob created. Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105102\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pipeline_job = aiplatform.PipelineJob.get('projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105102')\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:pipeline_job = aiplatform.PipelineJob.get('projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105102')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/custom-training-pipeline-20221014105102?project=21417957250\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/custom-training-pipeline-20221014105102?project=21417957250\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Associating projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105102 to Experiment: qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.metadata.experiment_resources:Associating projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105102 to Experiment: qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:Creating PipelineJob\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob created. Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105105\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob created. Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105105\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pipeline_job = aiplatform.PipelineJob.get('projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105105')\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:pipeline_job = aiplatform.PipelineJob.get('projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105105')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/custom-training-pipeline-20221014105105?project=21417957250\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/custom-training-pipeline-20221014105105?project=21417957250\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Associating projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105105 to Experiment: qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.metadata.experiment_resources:Associating projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105105 to Experiment: qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:Creating PipelineJob\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob created. Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105108\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob created. Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105108\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pipeline_job = aiplatform.PipelineJob.get('projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105108')\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:pipeline_job = aiplatform.PipelineJob.get('projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105108')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/custom-training-pipeline-20221014105108?project=21417957250\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/custom-training-pipeline-20221014105108?project=21417957250\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Associating projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105108 to Experiment: qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.metadata.experiment_resources:Associating projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105108 to Experiment: qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:Creating PipelineJob\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob created. Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105111\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob created. Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105111\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pipeline_job = aiplatform.PipelineJob.get('projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105111')\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:pipeline_job = aiplatform.PipelineJob.get('projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105111')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/custom-training-pipeline-20221014105111?project=21417957250\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/custom-training-pipeline-20221014105111?project=21417957250\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Associating projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105111 to Experiment: qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.metadata.experiment_resources:Associating projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105111 to Experiment: qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    }
   ],
   "source": [
    "# TODO 3\n",
    "for i, run in enumerate(runs):\n",
    "\n",
    "    job = vertex_ai.PipelineJob(\n",
    "        display_name=f\"{EXPERIMENT_NAME}-pipeline-run-{i}\",\n",
    "        template_path=PIPELINE_TEMPLATE_FILE,\n",
    "        pipeline_root=PIPELINE_URI,\n",
    "        parameter_values={\n",
    "            \"train_uri\": TRAIN_URI,\n",
    "            \"label_uri\": LABEL_URI,\n",
    "            \"model_uri\": MODEL_URI,\n",
    "            **run,\n",
    "        },\n",
    "    )\n",
    "    # Submit the pipeline job using job.submit()\n",
    "    # TODO 3: Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "O8TV4q535c2M"
   },
   "source": [
    "### Check Pipeline run states\n",
    "\n",
    "Vertex AI SDK provides you `get_experiment_df` method to monitor the status of pipeline runs. You can use it either to return parameters and metrics of the Pipeline Runs in the Vertex AI Experiment or in combination with `get` method of `PipelineJob` to return the pipeline job in Vertex AI Pipeline.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "dlCEJKfH5xR7"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>experiment_name</th>\n",
       "      <th>run_name</th>\n",
       "      <th>run_type</th>\n",
       "      <th>state</th>\n",
       "      <th>param.max_depth</th>\n",
       "      <th>param.boost_rounds</th>\n",
       "      <th>param.learning_rate</th>\n",
       "      <th>param.label_uri</th>\n",
       "      <th>param.model_uri</th>\n",
       "      <th>param.train_uri</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>qwiklabs-gcp-00-1fe7a1835a07-classification-xg...</td>\n",
       "      <td>custom-training-pipeline-20221014105111</td>\n",
       "      <td>system.PipelineRun</td>\n",
       "      <td>RUNNING</td>\n",
       "      <td>5</td>\n",
       "      <td>30</td>\n",
       "      <td>0.4</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_ta...</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/model</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_da...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>qwiklabs-gcp-00-1fe7a1835a07-classification-xg...</td>\n",
       "      <td>custom-training-pipeline-20221014105108</td>\n",
       "      <td>system.PipelineRun</td>\n",
       "      <td>RUNNING</td>\n",
       "      <td>6</td>\n",
       "      <td>40</td>\n",
       "      <td>0.5</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_ta...</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/model</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_da...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>qwiklabs-gcp-00-1fe7a1835a07-classification-xg...</td>\n",
       "      <td>custom-training-pipeline-20221014105105</td>\n",
       "      <td>system.PipelineRun</td>\n",
       "      <td>RUNNING</td>\n",
       "      <td>3</td>\n",
       "      <td>30</td>\n",
       "      <td>0.1</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_ta...</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/model</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_da...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>qwiklabs-gcp-00-1fe7a1835a07-classification-xg...</td>\n",
       "      <td>custom-training-pipeline-20221014105102</td>\n",
       "      <td>system.PipelineRun</td>\n",
       "      <td>RUNNING</td>\n",
       "      <td>5</td>\n",
       "      <td>20</td>\n",
       "      <td>0.3</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_ta...</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/model</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_da...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>qwiklabs-gcp-00-1fe7a1835a07-classification-xg...</td>\n",
       "      <td>custom-training-pipeline-20221014105059</td>\n",
       "      <td>system.PipelineRun</td>\n",
       "      <td>RUNNING</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>0.2</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_ta...</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/model</td>\n",
       "      <td>gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_da...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     experiment_name  \\\n",
       "0  qwiklabs-gcp-00-1fe7a1835a07-classification-xg...   \n",
       "1  qwiklabs-gcp-00-1fe7a1835a07-classification-xg...   \n",
       "2  qwiklabs-gcp-00-1fe7a1835a07-classification-xg...   \n",
       "3  qwiklabs-gcp-00-1fe7a1835a07-classification-xg...   \n",
       "4  qwiklabs-gcp-00-1fe7a1835a07-classification-xg...   \n",
       "\n",
       "                                  run_name            run_type    state  \\\n",
       "0  custom-training-pipeline-20221014105111  system.PipelineRun  RUNNING   \n",
       "1  custom-training-pipeline-20221014105108  system.PipelineRun  RUNNING   \n",
       "2  custom-training-pipeline-20221014105105  system.PipelineRun  RUNNING   \n",
       "3  custom-training-pipeline-20221014105102  system.PipelineRun  RUNNING   \n",
       "4  custom-training-pipeline-20221014105059  system.PipelineRun  RUNNING   \n",
       "\n",
       "  param.max_depth param.boost_rounds  param.learning_rate  \\\n",
       "0               5                 30                  0.4   \n",
       "1               6                 40                  0.5   \n",
       "2               3                 30                  0.1   \n",
       "3               5                 20                  0.3   \n",
       "4               4                 10                  0.2   \n",
       "\n",
       "                                     param.label_uri  \\\n",
       "0  gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_ta...   \n",
       "1  gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_ta...   \n",
       "2  gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_ta...   \n",
       "3  gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_ta...   \n",
       "4  gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_ta...   \n",
       "\n",
       "                           param.model_uri  \\\n",
       "0  gs://qwiklabs-gcp-00-1fe7a1835a07/model   \n",
       "1  gs://qwiklabs-gcp-00-1fe7a1835a07/model   \n",
       "2  gs://qwiklabs-gcp-00-1fe7a1835a07/model   \n",
       "3  gs://qwiklabs-gcp-00-1fe7a1835a07/model   \n",
       "4  gs://qwiklabs-gcp-00-1fe7a1835a07/model   \n",
       "\n",
       "                                     param.train_uri  \n",
       "0  gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_da...  \n",
       "1  gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_da...  \n",
       "2  gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_da...  \n",
       "3  gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_da...  \n",
       "4  gs://qwiklabs-gcp-00-1fe7a1835a07/iris/iris_da...  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# see state of all pipelineJob\n",
    "vertex_ai.get_experiment_df(EXPERIMENT_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "98c022ca36b4"
   },
   "source": [
    "The pipeline runs in the Vertex AI Experiment will be monitored based on pipeline run status."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "FA9W85vs7LLD"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline runs are still running...\n",
      "Pipeline runs are still running...\n",
      "Pipeline experiment runs have completed\n"
     ]
    }
   ],
   "source": [
    "while True:\n",
    "    pipeline_experiments_df = vertex_ai.get_experiment_df(EXPERIMENT_NAME)\n",
    "    if any(\n",
    "        pipeline_state != \"COMPLETE\" for pipeline_state in pipeline_experiments_df.state\n",
    "    ):\n",
    "        print(\"Pipeline runs are still running...\")\n",
    "        if any(\n",
    "            pipeline_state == \"FAILED\"\n",
    "            for pipeline_state in pipeline_experiments_df.state\n",
    "        ):\n",
    "            print(\"At least one Pipeline run failed\")\n",
    "            break\n",
    "    else:\n",
    "        print(\"Pipeline experiment runs have completed\")\n",
    "        break\n",
    "    time.sleep(60)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "ISsK9Msi-Kqs"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline job name:  projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105111\n",
      "Pipeline Run UI link:  https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/custom-training-pipeline-20221014105111?project=21417957250\n"
     ]
    }
   ],
   "source": [
    "# TODO 4\n",
    "# Get the PipelineJob resource using the experiment run name\n",
    "pipeline_experiments_df = vertex_ai.get_experiment_df(EXPERIMENT_NAME)\n",
    "job = # TODO 4: Your code goes here\n",
    "print(\"Pipeline job name: \", job.resource_name)\n",
    "print(\"Pipeline Run UI link: \", job._dashboard_uri())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TpV-iwP9qw9c"
   },
   "source": [
    "## Cleaning up\n",
    "\n",
    "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
    "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
    "\n",
    "Otherwise, you can delete the individual resources you created in this tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "6xbYQn5t5Noe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline job is complete.\n",
      "Deleting PipelineJob : projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105111\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting PipelineJob : projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105111\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete PipelineJob  backing LRO: projects/21417957250/locations/us-central1/operations/5165028782842576896\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete PipelineJob  backing LRO: projects/21417957250/locations/us-central1/operations/5165028782842576896\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob deleted. . Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105111\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:PipelineJob deleted. . Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105111\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline job is complete.\n",
      "Deleting PipelineJob : projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105108\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting PipelineJob : projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105108\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete PipelineJob  backing LRO: projects/21417957250/locations/us-central1/operations/7175323066510082048\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete PipelineJob  backing LRO: projects/21417957250/locations/us-central1/operations/7175323066510082048\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob deleted. . Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105108\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:PipelineJob deleted. . Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105108\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline job is complete.\n",
      "Deleting PipelineJob : projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105105\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting PipelineJob : projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105105\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete PipelineJob  backing LRO: projects/21417957250/locations/us-central1/operations/2957139065524191232\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete PipelineJob  backing LRO: projects/21417957250/locations/us-central1/operations/2957139065524191232\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob deleted. . Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105105\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:PipelineJob deleted. . Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105105\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline job is complete.\n",
      "Deleting PipelineJob : projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting PipelineJob : projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105102\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete PipelineJob  backing LRO: projects/21417957250/locations/us-central1/operations/7568825083951579136\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete PipelineJob  backing LRO: projects/21417957250/locations/us-central1/operations/7568825083951579136\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob deleted. . Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:PipelineJob deleted. . Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105102\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline job is complete.\n",
      "Deleting PipelineJob : projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105059\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting PipelineJob : projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105059\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete PipelineJob  backing LRO: projects/21417957250/locations/us-central1/operations/5180228431584952320\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete PipelineJob  backing LRO: projects/21417957250/locations/us-central1/operations/5180228431584952320\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob deleted. . Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105059\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:PipelineJob deleted. . Resource name: projects/21417957250/locations/us-central1/pipelineJobs/custom-training-pipeline-20221014105059\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting Context : projects/21417957250/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting Context : projects/21417957250/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete Context  backing LRO: projects/21417957250/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq/operations/3271828089486704640\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete Context  backing LRO: projects/21417957250/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq/operations/3271828089486704640\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context deleted. . Resource name: projects/21417957250/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Context deleted. . Resource name: projects/21417957250/locations/us-central1/metadataStores/default/contexts/qwiklabs-gcp-00-1fe7a1835a07-classification-xgboost-406gb4wq\n"
     ]
    }
   ],
   "source": [
    "# Delete the pipeline\n",
    "while True:\n",
    "    for i in range(0, len(runs)):\n",
    "        pipeline_job = vertex_ai.PipelineJob.get(pipeline_experiments_df.run_name[i])\n",
    "        if pipeline_job.state != PipelineState.PIPELINE_STATE_SUCCEEDED:\n",
    "            print(\"Pipeline job is still running...\")\n",
    "            time.sleep(60)\n",
    "        else:\n",
    "            print(\"Pipeline job is complete.\")\n",
    "            pipeline_job.delete()\n",
    "    break\n",
    "\n",
    "# Delete experiment\n",
    "exp = vertex_ai.Experiment(EXPERIMENT_NAME)\n",
    "exp.delete()\n",
    "\n",
    "# Delete bucket\n",
    "delete_bucket = False\n",
    "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
    "    ! gsutil rm -rf {BUCKET_URI}\n",
    "\n",
    "# Remove local files\n",
    "\n",
    "!rm {PIPELINE_TEMPLATE_FILE}"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "comparing_pipeline_runs.ipynb",
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m97",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m97"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
